Repository navigation
Keep a managed Worker running through a transient Server or network outage (#448) - #449
Merged
Merged
Conversation
…utage Fixes #448. A Worker process died when native Server answered a workflow poll with 503 {"reason":"storage_unavailable"} while its database warmed up: pollFailurePolicy() retried only typed refusals, and Worker::run() rethrew everything except a stale registration. - Worker\TransientFailure classifies untyped transients: a connection that is refused, reset or timed out; an upstream 502/503/504 with no body; a 503 storage_unavailable or backend_unavailable; and any 5xx or 429 with retryable: true. Never retryable: false, auth failures or other 4xx. - Polls retry them and keep the poll identity (an outage committed nothing), using the shared jittered schedule: 0.25 s doubling to a 4 s cap, full jitter. Typed refusals keep their protocol backoff and identity rules. - Worker::run() absorbs a transient failure that escapes a tick (a completion, for example) with the same schedule, logging worker.transient_failure, and keeps heartbeating while it waits. Fatal failures log worker.fatal_failure with status and reason before run() rethrows. - Tests use a fake transport: 503s then recovery leaves the Worker running; a failure escaping a tick is absorbed; 401, 403, 400 and retryable: false stop the Worker with one fatal diagnostic; classification and jitter bounds. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Coordinator decisions on #448: - A completion that meets an untyped outage (503 storage_unavailable or backend_unavailable) is retried with the same attempt fence on the shared jittered schedule, like the existing connection-failure retry. With activity lease maintenance the publication fence ends retries at the grant's end; without it, eight attempts do (worker.outcome_retry_exhausted). - Idempotent: verified against published Server 3.17.0. The first completion committed, the Worker saw a 503, and the retry was answered 409 stale_attempt, recorded false, attempt completed. The existing isCommittedActivityCompletion() rule now applies to it, so the run completed with one ActivityCompleted and the activity ran once. - A computed result whose publication an outage defeats is logged (worker.outcome_unpublished) and left to the Server's lease handling, never sent as an activity failure report. - An unclassified retryable answer still never authorizes repeating an acknowledgement, and typed refusals keep their own protocol checks, so an invalid typed retry_after cannot stall the Worker. Server-suggested delays for untyped outages are capped at 60 s. - Docs: completion retry, and the cross-SDK difference (PHP and Rust stop on 401/403 after an error; Python retries polls so credentials can rotate). Co-Authored-By: Claude Opus 5.5 <[email protected]>
…answers Review HIGHs on #449, found live against Server 3.17.0 through a fault proxy: 1. Registration (the original #448 incident) retried only typed refusals, so 503 storage_unavailable during warm-up still ended the Worker. It now uses the shared transient classification and jittered backoff, and so do the cluster-info reads registration makes in query-authority and cooperative modes. 2. The worker heartbeat inside a retry wait rethrew an untyped outage, so a 12 s outage after registration ended run(). The heartbeat now defers it on the shared schedule like its other transient failures, so no network call in the wait path can end the run. 3. cURL can resend a completion whose reply was lost on a reused connection, so the first answer the Worker sees may already be the Server's duplicate: 409 stale_attempt, recorded false, our owner and attempt, every status completed. That exact shape is accepted as our committed completion on the first answer (worker.completion_already_committed), not only after a seen failure. Tests reproduce all three with a fake transport and fail on the previous head. A live check against durableworkflow/server:3.17.0 with injected faults (three registration 503s, then a 12 s outage of polls and heartbeats) shows no Worker error, and workflows started before and after both complete. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The registration retry also retried connection failures, so the existing contract that an ambiguous registration (the Server may have registered) is not blindly repeated looped until the CI timeout. Registration now retries only the Server's definite outage answer, 503 storage_unavailable or backend_unavailable (the #448 warm-up incident), on the shared jittered schedule. Full suite: 9216 tests, 0 failures. Co-Authored-By: Claude Opus 5.5 <[email protected]>
rmcdaniel
added a commit
that referenced
this pull request
Nov 20, 2026
Digest re-review: - BLOCKER: a garbled 200 to an activity completion triggered the no-digest resend, the Server answered 409 stale_attempt for the completion it had already committed, and Worker::run exited. The committed-completion rule (409 stale_attempt, recorded false, our owner and attempt, every status completed) now applies to any answer: the first, after a transient failure, or after a digest resend. This block is identical to the one in #449, so whichever merges second merges cleanly. Workflow-task duplicates need nothing: Server 3.17.0 answers them 200 recorded:false already_completed, which the Worker already accepts. - End-to-end test: garbled completion answer, then the resend's 409; the Worker keeps running and the activity runs once. Verified live against durableworkflow/server:3.17.0 as well. - The default decode limit is min(32 MiB, memory_limit / 4), safe under PHP's default 128 MB, and ResponseTooLarge reaches callers unwrapped. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member
Author
|
The committed-completion block in |
Re-check follow-up on #449: a Worker started during an outage still exited on a refused connection, an empty reply or a bodiless 502/503/504 at registration. - Registration retries every outage on the shared jittered schedule: refused or reset connections, empty replies, bodiless 502/503/504 from a load balancer, and 503 storage_unavailable / backend_unavailable. Registration is idempotent: against durableworkflow/server:3.17.0, registering the same worker ID twice answers registered: true both times. The two tests that forbade the repeat are replaced by ones asserting it keeps the original identity. - An outage lasting 60 s is reported once as a worker.outage_persisting error, then every 5 minutes, until a request succeeds (registration, poll retries and the run loop). - Tests: server down at startup (refused connection, empty reply), a load balancer answering bare 502 and 504, a bodiless 503 behind a load balancer, and the 60 s / 5 min escalation on a simulated clock. Live: a Worker started with the Server stopped for about 15 s registered once it returned and completed a workflow. Full suite: 9221 tests, 0 failures. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
rmcdaniel
added a commit
that referenced
this pull request
Nov 20, 2026
Digest re-review: - BLOCKER: a garbled 200 to an activity completion triggered the no-digest resend, the Server answered 409 stale_attempt for the completion it had already committed, and Worker::run exited. The committed-completion rule (409 stale_attempt, recorded false, our owner and attempt, every status completed) now applies to any answer: the first, after a transient failure, or after a digest resend. This block is identical to the one in #449, so whichever merges second merges cleanly. Workflow-task duplicates need nothing: Server 3.17.0 answers them 200 recorded:false already_completed, which the Worker already accepts. - End-to-end test: garbled completion answer, then the resend's 409; the Worker keeps running and the activity runs once. Verified live against durableworkflow/server:3.17.0 as well. - The default decode limit is min(32 MiB, memory_limit / 4), safe under PHP's default 128 MB, and ResponseTooLarge reaches callers unwrapped. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Fixes #448.
Problem
A managed Worker process died when native Server answered a workflow poll with
503 {"reason":"storage_unavailable","message":"storage_unavailable"}while its database warmed up. Two gaps combined:Worker::pollFailurePolicy()retried a 503 only when it was a typed poll refusal or a typed backend/storage-admission refusal.Worker::run()rethrew everything except a stale-registration recovery.Evidence is in #448: the lab run with Server main
43ad8d8fand SDK 3.22.0, whereworker-3.logends with an uncaughtTransportException.Change
Worker\TransientFailureclassifies untyped transients:storage_unavailableorbackend_unavailable;retryable: true.It never matches
retryable: false, 401 or 403, or other 4xx protocol refusals.Polls retry these and keep the poll identity, since an outage committed nothing. The delay follows the schedule all SDKs share: 0.25 s doubling to a 4 s cap, with full random jitter, and never shorter than a
retry_after_secondsthe Server sends. Typed refusals keep their existing protocol backoff and identity rules; a test guards the lock-pressure identity rule.Worker::run()absorbs a transient failure that escapes a tick (a completion, for example) on the same schedule. It logsworker.transient_failure, keeps heartbeating while it waits, and continues. The Server re-delivers any claimed task when its lease ends. Fatal failures logworker.fatal_failurewith status and reason beforerun()rethrows.Docs and changelog:
docs/portal/build/workers.mdandCHANGELOG.mddescribe the behaviour.Completions that meet an untyped outage are retried with the same attempt fence on the shared schedule.
worker.outcome_retry_exhausted).worker.outcome_unpublishedand left to the Server's lease handling. It is never sent as an activity failure report.Unchanged: an unclassified
retryable: trueanswer still never authorizes repeating an acknowledgement (the existing contract tests inWorkerStorageAdmissionTestandWorkerWorkflowCompletionLockPressureTest). Typed refusals keep their own protocol checks, and Server-suggested delays for untyped outages are capped at 60 s.Review fixes (live fault-proxy findings)
503 storage_unavailable/backend_unavailable. The cluster-info reads registration makes in query-authority and cooperative modes do the same, so a Worker started while the Server is down or warming up waits for it.durableworkflow/server:3.17.0, registering the same worker ID twice returnsregistered: trueboth times.WorkerConnectionRecoveryTest::testAmbiguousRegistrationIsNotRepeatedandWorkerUpstreamRecoveryTest::testRegistrationDoesNotBlindlyRepeatAnUncertainRequest) are replaced. The new tests assert the repeat keeps the original identity.worker.outage_persistingerror, then every 5 minutes until a request succeeds.run().stale_attempt,recorded: false, our owner and attempt, every statuscompleted. That exact shape is accepted as our committed completion on the first answer (worker.completion_already_committed).All three are reproduced by fake-transport tests that fail on the previous head.
Live check, Worker started with the Server down: the Server container was stopped for about 15 s at Worker start. The Worker retried 13 times, registered once the Server was back, and completed a workflow started afterwards. There was no Worker error and no fatal diagnostic.
Live check against
durableworkflow/server:3.17.0, with faults injected by a Guzzle middleware:completed.Idempotency against published Server 3.17.0
This is a real Worker with a Guzzle middleware that forwards the first activity completion to
durableworkflow/server:3.17.0. The Server commits it, and the middleware then shows the Worker503 storage_unavailable.200 recorded: true(committed, the reply hidden from the Worker)409 {"outcome":"completed","recorded":false,"reason":"stale_attempt","activity_status":"completed","attempt_status":"completed","task_status":"completed",…}isCommittedActivityCompletion)completed, oneActivityCompleted, noActivityFailed, activity ran oncePython (#355) and Rust (#305) observed the same answer and apply the same rule.
Across SDKs
PHP and Rust stop on a 401 or 403 after logging an error. Python keeps retrying polls through a 401 or 403 so credentials can rotate, and logs every refusal at error level. This is documented in each SDK.
Tests (
tests/WorkerTransientOutageTest.php, using a fake transport)main503 storage_unavailablepolls, then recoveryworker.transient_failureis logged and the Worker keeps pollingretryable: falseworker.fatal_failurediagnosticworker.outcome_unpublished; no failure reportResults:
phpstan: no errors.🤖 Generated with Claude Code